Image processing method and apparatus

ABSTRACT

Provided are an image processing apparatus and an image processing method. The image processing method includes: obtaining a reference block a having a first resolution and a differential block having the first resolution; transforming a resolution of a prediction block having the first resolution and a resolution of the differential block having the first resolution in order to generate a prediction block having a second resolution and a differential block having the second resolution, the prediction block having the first resolution and the differential block having the first resolution being generated based on a pixel value of the reference block having the first resolution; and synthesizing the generated prediction block having the second resolution and the generated differential block having the second resolution in order to generate a target block having the second resolution.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Patent Application No.10-2010-0042506, filed on May 6, 2010 in the Korean IntellectualProperty Office, the entire disclosure of which is incorporated hereinby reference.

BACKGROUND

1. Field

Methods and apparatuses consistent with example embodiments relate toconverting a resolution of a compressed image.

2. Description of the Related Art

When displaying a compressed image through transformation of aresolution, the compressed image is decoded according to decodingformats corresponding to encoding formats defined by differentstandards. Then, a downsizing process to decrease a resolution of thedecoded image or an upsizing process to increase the resolution of thedecoded image is carried out according to a resolution of a displaydevice to reproduce the image.

SUMMARY

According to an aspect of an exemplary embodiment, there is provided animage processing method including: obtaining a reference block having afirst resolution and a differential block having the first resolution,the referential block being defined based on a difference between thereference block and a target block having the first resolution;transforming a resolution of a prediction block having the firstresolution and a resolution of the differential block having the firstresolution in order to generate a prediction block having a secondresolution and a differential block having the second resolution, theprediction block having the first resolution and the differential blockhaving the first resolution being generated based on a pixel value ofthe reference block having the first resolution; and synthesizing thegenerated prediction block having the second resolution and thegenerated differential block having the second resolution in order togenerate a target block having the second resolution.

According to an aspect of another exemplary embodiment, there isprovided an image processing method including: obtaining a referenceblock a having a first resolution and a differential block having thefirst resolution, the referential block being defined based on adifference between the reference block and a target block having thefirst resolution; transforming a resolution of a prediction block havingthe first resolution and a resolution of the differential block havingthe first resolution in order to generate a prediction block having asecond resolution and a differential block having the second resolution,the prediction block having the first resolution and the differentialblock having the first resolution being generated based on a pixel valueof an adjacent area to the reference block having the first resolution;and synthesizing the generated prediction block having the secondresolution and the generated differential block having the secondresolution in order to generate a target block having the secondresolution.

According to an aspect of another exemplary embodiment, there isprovided an image processing method including: transforming a magnitudeof a motion vector of a target block having a first resolution at apredetermined ratio in order to generate a transformed motion vector;generating a prediction block having a second resolution based on apixel value of a reference block having the second resolution indicatedby the transformed motion vector; transforming a resolution of adifferential block having the first resolution in order to generate adifferential block having the second resolution, the differential blockhaving the first resolution defined based on a difference between areference block having the first resolution and the target block havingthe first resolution; and synthesizing the generated prediction blockhaving the second resolution and the generated differential block havingthe second resolution in order to generate a target block having thesecond resolution.

According to an aspect of another exemplary embodiment, there isprovided an image processing method including: transforming a magnitudeof a motion vector of a target block having a first resolution at apredetermined ratio in order to generate a transformed motion vector;generating a prediction block having a second resolution based on apixel value of an adjacent area to a reference block having the secondresolution indicated by the transformed motion vector; transforming aresolution of a differential block having the first resolution in orderto generate a differential block having the second resolution, thedifferential block having the first resolution defined based on adifference between a reference block having the first resolution and thetarget block having the first resolution; and synthesizing the generatedprediction block having the second resolution and the generateddifferential block having the second resolution in order to generate atarget block having the second resolution.

According to an aspect of another exemplary embodiment, there isprovided an image processing apparatus including: a first transformationunit which transforms a differential block having a first resolutioninto a differential block having a second resolution in order togenerate the differential block having the second resolution; a secondtransformation unit which obtains a prediction block having the firstresolution generated based on a pixel value of a reference block havingthe first resolution and transforms the prediction block into aprediction block having the second resolution in order to generate theprediction block having the second resolution; and a synthesis unitwhich synthesizes the generated prediction block having the secondresolution and the generated differential block having the secondresolution in order to generate a target block having the secondresolution.

According to an aspect of another exemplary embodiment, there isprovided an image processing apparatus including: a motion vectortransformation unit which transforms a magnitude of a motion vector of atarget block having a first resolution at a predetermined ratio in orderto generate a transformed motion vector; a second transformation unitwhich generates a prediction block having a second resolution based on apixel value of a reference block having the second resolution indicatedby the transformed motion vector; a first transformation unit whichtransforms a resolution of a differential block having the firstresolution in order to generate a differential block having the secondresolution, the differential block having the first resolution definedbased on a difference between a reference block having the firstresolution and the target block having the first resolution; and asynthesis unit which synthesizes the generated prediction block havingthe second resolution and the generated differential block having thesecond resolution in order to generate a target block having the secondresolution.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects will become apparent and more readilyappreciated from the following detailed description of certain exemplaryembodiments, taken in conjunction with the accompanying drawings ofwhich:

FIG. 1 illustrates an image interpolation process according to a relatedart;

FIG. 2 is a block diagram illustrating an image processing apparatusaccording to an exemplary embodiment;

FIG. 3 illustrates motion compensation and resolution transformationprocesses according to an exemplary embodiment;

FIG. 4 illustrates an example operation of a second transformation unitof FIG. 2 performing motion compensation and resolution transformationof an I-frame;

FIG. 5 illustrates an example operation of the second transformationunit of FIG. 2 performing motion compensation and resolutiontransformation of a P-frame;

FIG. 6 illustrates an example operation of the second transformationunit of FIG. 2 performing motion compensation and resolutiontransformation of an Other P-frame;

FIG. 7 illustrates a configuration of a first transformation unitaccording to an exemplary embodiment;

FIG. 8 illustrates a configuration of a second transformation unitaccording to an exemplary embodiment;

FIG. 9 is a block diagram illustrating a decoding mode determinationapparatus according to an exemplary embodiment;

FIG. 10 illustrates an example of determining a decoding mode;

FIG. 11 is a flowchart illustrating an image decoding process accordingto an exemplary embodiment;

FIG. 12 is a flowchart illustrating an image decoding process accordingto another exemplary embodiment; and

FIG. 13 is a flowchart illustrating a decoding mode determinationprocess according to an exemplary embodiment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Reference will now be made in detail to exemplary embodiments, examplesof which are illustrated in the accompanying drawings, wherein likereference numerals refer to the like elements throughout. Exemplaryembodiments are described below in order to explain the presentdisclosure by referring to the figures. Hereinafter, expressions such as“at least one of,” when preceding a list of elements, modify the entirelist of elements and do not modify the individual elements of the list.

When decoding a compressed image, inter-prediction or intra-predictionis applicable to each of a plurality of blocks to form one frameaccording to a compression mode.

Intra-prediction determines a pixel value of a target block using apixel value of an adjacent block similar to the target block in theframe being decoded. Inter-prediction determines a pixel value of atarget block using a motion vector of the target block in a frame beingdecoded. In further detail, inter-prediction extracts a reference blockcorresponding to the target block from a reference frame positionedbefore or after the frame being decoded and generates a prediction blockusing a pixel value and a motion vector of the reference block. Theprediction block is used to determine a pixel value of the target block.

Intra- and inter-prediction may be used to reconstruct an I-frame or aP-frame. Each block in an I-frame may be compressed and reconstructed inone frame according to connection between blocks using theintra-prediction mode. At least one of intra-prediction andinter-prediction may be applied to each block in a P-frame based onproperties of the frame. When using an inter-prediction mode, motioncompensation may be implemented based on an I-frame or a P-frame inorder to reconstruct another P-frame.

Motion compensation implemented in inter-prediction obtains a pixelvalue for reconstruction from at least one reference frame positionedbefore or after a frame being coded. Motion compensation may beimplemented by an integer pixel unit, or may be implemented by afractional pixel unit having greater precision than an integer pixel inorder to increase accuracy of prediction. For example, motioncompensation may be implemented using a pixel value of a ½ pixel(half-pel) and a ¼ pixel (quarter-pel) interpolated using an integerpixel of a reference frame.

FIG. 1 illustrates a process of interpolating an image according to arelated art.

In FIG. 1, capital letters represent integer pixels, lowercase lettersin grey represent ½ pixels (half-pets), and lowercase letters in whiterepresent ¼ pixels (quarter-pels).

Referring to FIG. 1, a ½ pixel positioned between two adjacent integerpixels in a horizontal direction is generated by applying a 6-tap finiteimpulse response (FIR) filter horizontally. For example, a ½ pixel b 120positioned between horizontal integer pixels C3 110 and C4 130 isgenerated by the following equation c=(C1−5C2+20C3+20C4−5C5+C6)/32 byapplying the 6-tap FIR filter to six adjacent integer pixels. In theabove equation, allocated values 1, −5, 20, 20, −5, and −1 to the sixinteger pixels adjacent to the interpolated ½ pixel are referred to as afilter coefficient.

Similarly, a ½ pixel positioned between two neighboring integer pixelsin a vertical direction is generated by applying the 6-tap finiteimpulse response (FIR) filter vertically. For example, a ½ pixel h 140positioned between vertical integer pixels C3 110 and D3 150 isgenerated by the following equation c=(A3−5B3+20C3+20D3−5E3+F3)/32 byapplying the 6-tap FIR filter to six adjacent integer pixels. ¼ pixelspositioned between two ½ pixels are also calculated by averagingadjacent integer pixels or ½ pixels in a similar manner.

A motion-compensated prediction block is synthesized with a differentialblock to reconstruct an image with the same resolution as a resolutionof an original image. The differential block is an image constituted bya difference value between a reference block located in a referenceframe and the original image, and is obtained by performinginverse-quantizing and inverse-transforming of a differential signalobtained from a received bitstream.

A decoded image is provided to a display device supporting a lowresolution via a downsizing process to lower a resolution. A downsizingprocess is implemented by applying a bilinear filter or Wiener filter. Adownsizing process may be performed by a process in which pixels usedfor resolution transformation are located, multiplied by a predeterminedvalue, and summated. When applying a bilinear filter, an average valueof two pixels is used to determine a pixel value used for a lowresolution. When applying a Wiener filter, a plurality of pixels aremultiplied by a filter coefficient obtained using a sync operation and aresolution-transformed pixel and summated, thereby obtainingresolution-transformed pixels.

A resolution transformation process of a compressed image according torelated schemes may be performed by a process in which a compressedimage is decoded and a resolution of the decoded image is lowered. Theresolution transformation process may have an effect on a device havinglimited resources to decode a compressed image and transform aresolution of the decoded image.

FIG. 2 is a block diagram illustrating an image processing apparatus 200according to an exemplary embodiment.

Referring to FIG. 2, an image decoding apparatus 200 includes an entropydecoder 210, a rearrangement unit 220, an inverse quantization unit 230,a first transformation unit 250, a second transformation unit 260, adeblocking unit 280, a motion vector transformation unit 240, and asynthesis unit 270.

The entropy decoder 210 receives a compressed bitstream and performsentropy decoding to generate quantized coefficients, and extracts motionvector information of a decoded target block.

The rearrangement unit 220 rearranges quantized coefficientsaccommodated by the entropy decoder 210.

The inverse quantization unit 230 inverse-quantizes rearranged quantizedcoefficients to reconstruct a differential signal before the inversequantization.

The first transformation unit 250 generates a differential block havinga second resolution. In further detail, the first transformation unit250 may accommodate a differential block having a first resolution togenerate the differential block having the second resolution.

Here, conceptual operations of the first transformation unit 250 includeat least one of a resolution transformation 710 and an inversetransformation 720 as in a first transformation unit 770 of FIG. 7. Theinverse transformation 720 and the resolution transformation 710 may beperformed sequentially or in parallel.

The inverse transformation 720 of the first transformation unit 250extracts a signal formed of frequency values from the inversequantization unit 230 and inverse-transforms the extracted signal togenerate a differential image of transformed spatial values. Thegenerated differential image is an image constituted by a differencevalue between a decoded target block and a reference block. Theresolution transformation 710 of the first transformation unit 250transforms a resolution of the generated differential image to generatea differential block by using an inverse transformation and a resolutiontransformation.

A resolution transformation process is performed by downsizing to lowera resolution or upsizing to increase a resolution. The resolutiontransformation process may be performed by applying a bilinear filter orWiener filter.

When applying a bilinear filter to decrease a resolution, an averagevalue of two pixels in a differential image may be used as a pixel valueof a differential block with a low resolution. When applying a Wienerfilter, a plurality of pixels are multiplied by a filter coefficientobtained using a sync operation and summated, thereby determining apixel value of a differential block with a low resolution.

Also, when applying a bilinear filter to increase a resolution, anaverage value of two pixels in a differential image may be interpolatedbetween the pixels to generate an additional pixel value for a higherresolution. When applying a Wiener filter, a plurality of pixels aremultiplied by a filter coefficient obtained using a sync operation andsummated. The summated pixels are interpolated between the pixels,thereby determining a pixel value for a higher resolution.

In addition to a bilinear filter or Wiener filter, various filters suchas a b-spline filter may be used to transform a resolution. A method ofapplying a filter is not limited to the above examples and may vary.

The second transformation unit 260 generates a prediction block havingthe second resolution. In further detail, the second transformation unit260 may accommodate a prediction block having the first resolution togenerate the prediction block having the second resolution.

Here, conceptual operation of the second transformation unit 260 includeat least one of a resolution transformation 810, a motion compensation820, and an intra-prediction 830 as in a second transformation unit 260of FIG. 8.

The resolution transformation 810, the motion compensation 820, and theintra-prediction 830 may be performed sequentially, or two or threeoperations may be performed at the same time.

The intra-prediction 830 determines a pixel value of a target blockbased on a pixel value of an adjacent block to the target block in aframe being decoded.

The motion compensation 820 extracts a reference block corresponding toa target block from at least one reference frame positioned before orafter a frame being decoded using a motion vector of the target block inthe frame being decoded and performs motion compensation using a pixelvalue of the extracted reference block.

The resolution transformation 810 may be performed through downsizing tolower a resolution, or through upsizing to increase a resolution asdescribed above with reference to the first transformation unit 250, anda bilinear filter or Wiener filter may be applied for these processes. Amethod of applying these filters is not limited to the examplesdescribed in the example embodiment, and various filters may be used.

The second transformation unit 260 generates a prediction block to besynthesized with a differential block generated by the firsttransformation unit 250 using at least one of the resolutiontransformation 810, the motion compensation 820, and theintra-prediction 830. A method of generating a prediction block may varydepending on whether a frame to be decoded is an I-frame, P-frame, orOther P-frame with reference to other P-frames.

FIG. 4 illustrates an example operation of the second transformationunit 260 of FIG. 2 performing motion compensation and resolutiontransformation of an I-frame.

Referring to FIG. 4, the second transformation unit 260 extracts a pixelvalue of a target block in a frame 400 to be decoded from adjacentblocks and obtains the target block before resolution transformationusing the extracted pixel value. Each obtained block constitutes oneframe and generates a frame 410 with the second resolution via the aboveresolution transformation operation.

FIG. 5 illustrates an example operation of the second transformationunit 260 of FIG. 2 performing motion compensation and resolutiontransformation of a P-frame.

Referring to FIG. 5, the second transformation unit 260 extracts areference block indicated by a target block in a frame 520 to be decodedfrom an I-frame 510 in a front or a rear of the frame using a motionvector extracted from a bitstream. The second transformation unit 260performs motion compensation based on a pixel value of the extractedreference block to obtain a prediction block 530 having the firstresolution. The prediction block 530 having the first resolution istransformed into a prediction block 550 having the second resolutionthrough resolution transformation. The prediction block 550 having thesecond resolution is synthesized with a differential block 560 havingthe second resolution transformed from a differential block 540 havingthe first resolution to generate a target block having the secondresolution. The target block having the second resolution generates aP-frame 570 having the second resolution along with another targetblock.

The motion compensation operation of the second transformation unit 260may be performed by a fractional pixel unit more precise than an integerpixel in order to increase accuracy of prediction. For example, motioncompensation may be implemented using a pixel value of a ½ pixel(half-pel) and a ¼ pixel (quarter-pel) interpolated using an integerpixel of a reference frame.

Motion compensation of the second transformation unit by a fractionalpixel unit may be performed sequentially or at the same time along withresolution compensation. In order to perform motion compensation at thesame time with the resolution transformation operation, a combinationfilter is generated from a filter for motion compensation and a filterfor resolution transformation and applied to a pixel value of areference block. The following Table 1 illustrates an example of acombination filter to perform motion compensation and resolutiontransformation at the same time.

TABLE 1 MOTION KIND OF VECTOR FILTER RESOLUTION FILTER COEFFICIENTBILINEAR I {0, 0, 64, 64, 0, 0, 0} >> 7 (2-TAP) Q¹ {1, −4, 47, 72, 15,−4, 1} >> 7 H {2, −8, 30, 80, 30, −8, 2} >> 7 Q² {1, −4, 15, 72, 47, −4,1} >> 7 WIENER I {0, −0, 48, −72, 89, 382, 89, −72, 48, 0, 0, 0} >> 9FILTER Q¹ {1, −5, 46, −45, 18, 336, 190, −63, 16, 22, −5, 1} >> 9(7-TAP) H {2, −10, 44, −17, −54, 291, 291, −54, −17, 44, −10, 2} >> 9 Q²{1, −5, 22, 16, −63, 190, 336, 18, −45, 46, −5, 1} >> 9 WIENER I {0, 0,64, 0, −128, −96, 160, 608, 832, 608, 160, −96, −128, 0, 64, 0, 0} >> 11FILTER Q¹ {1, −5, 50, 29, −99, −120, 76, 494, 807, 695, 270, −53, −136,−35, 61, 18, −5, 1} >> 11 (12-TAP) H {2, −10, 36, 57, −70, −144, −9,380, 782, 782, 380, −9, −144, −70, 57, 36, −10, 2} >> 11 Q² {1, −5, 18,61, −35, −136, −53, 270, 695, 807, 494, 76, −120, −99, 29, 50, −5, 1} >>11

In Table 1, filter coefficients of a combination filter to performmotion compensation and resolution transformation at the same time maybe determined according to a type of filter for resolutiontransformation and a resolution of a motion vector needed for motioncompensation.

FIG. 3 illustrates a motion compensation process and a resolutiontransformation process according to an exemplary embodiment.

In FIG. 3, capital-letter pixels C3 310, C4 350, D3 390, and D4represent integer pixels, lowercase-letter pixels in grey b 330, h 370,j, i, and gg represent ½ pixels (half-pels), and lowercase-letter pixelsin white a 320, c 340, d 360, e, f, g, i, k, m 380, n, o, and prepresent ¼ pixels (quarter-pels).

Referring to FIG. 3, a combination filter in Table 1 is applied to aplurality of adjacent integer pixels in the horizontal or verticaldirection to generate a resolution-transformed pixel value. For example,when generating a resolution-transformed pixel value corresponding to aposition of a horizontal integer pixel C3 310 using a bilinear filter inTable 1, the pixel value may be generated by applying the combinationfilter to seven adjacent integer pixels to the integer pixel C3 310 bythe following equation c=(0·C1+0·C2+64C3+64C4+0·C5+0·C6+0·C7)>>7.

Similarly, in order to perform motion interpolation and resolutiontransformation by a ½ pixel unit at the same time, a plurality ofadjacent integer pixels in the horizontal or vertical direction may beapplied to a combination filter in Table 1. For example, when generatinga resolution-transformed pixel value corresponding to a position of a ½pixel b 330 positioned between the integer pixels C3 310 and C4 350using a bilinear filter in Table 1, the pixel value may be generated byapplying the combination filter to seven adjacent integer pixels by thefollowing equation b=(2C1−8C2+30C3+80C4+30C5−8C6+C2)>>7.

Similarly, in order to perform motion interpolation and resolutiontransformation by a ¼ pixel unit at the same time, a plurality ofadjacent integer pixels in the horizontal or vertical direction may beapplied to a combination filter in Table 1. For example, when generatinga resolution-transformed pixel value corresponding to a position of a ¼pixel a 320 positioned between the integer pixel C3 310 and thefractional pixel b 330 using a bilinear filter in Table 1, the pixelvalue may be generated by applying the combination filter to sevenadjacent integer pixels by the following equationa=(C1−4C2+47C3+72C4+15C5−4C6+C7)>>7. In the same manner, a ¼ pixel c 340positioned between b 330 and C4 350 may also be motion-interpolated andresolution-transformed at the same time by application of the abovecombination filter.

Although a bilinear filter or Wiener filter is given in Table 1 as anexample of a combination filter to perform motion compensation andresolution transformation at the same time, a type of filter accordingto another exemplary embodiment is not limited thereto, but may includeother filters or may vary. Further, a motion resolution may also not belimited to the present exemplary embodiment but may employ a pixel unithaving a different precision.

FIG. 6 illustrates an example operation of the second transformationunit 260 of FIG. 2 performing motion compensation and resolutiontransformation of Other P-frame.

Referring to FIG. 6, the second transformation unit 260 extracts areference block indicated by a target block in a frame 620 to be decodedfrom a P-frame in a front or a rear of the frame using a motion vectorextracted from a bitstream. Here, since the front or the rear P-frame610 is an already resolution-transformed frame, when using the extractedmotion vector as is for reference, a position of the reference blockcorresponding to the target block may not be detected accurately. Thus,a magnitude of the motion vector may need to be transformed through amotion vector transformation unit in order for the motion vector of thetarget block to accurately indicate the position of the reference block.The second transformation unit 260 performs motion compensation usingthe magnitude-changed motion vector through the motion vectortransformation unit 240 to generate a predication block 640. Thepredication block 640 having the second resolution is synthesized with adifferential block 650 having the second resolution transformed from adifferential block 630 having the first resolution to generate a targetblock having the second resolution. The target block having the secondresolution generates an Other P-frame 660 having the second resolutionalong with another target block.

The motion vector transformation unit 240 changes a magnitude of amotion vector extracted from a bitstream in proportion to a resolutionof an image to be displayed. When a resolution is high, a magnitude of amotion vector may increase proportionally. When a resolution is low, amagnitude of a motion vector may decrease proportionally.

The second transformation unit 260 extracts a reference block indicatedby a target block in a frame being decoded from a P-frame in a front ora rear of the frame using a motion vector changed in magnitude via themotion vector transformation unit 240. The second transformation unit260 performs motion compensation using a pixel value of the referenceblock to generate a prediction block.

Motion compensation of the second transformation unit 260 to generate aprediction block may be performed by a fractional pixel unit in order toincrease accuracy of prediction. Precise motion compensation may beimplemented using a pixel value of a ½ pixel (half-pel) and a ¼ pixel(quarter-pel) interpolated using adjacent pixels of a reference frame.

Here, motion compensation may be performed using a magnitude-transformedmotion vector. For example, when decreasing a resolution by half, amagnitude of a motion vector decreases by half. Thus, when a referenceblock is used to perform motion compensation by an integer pixel unit, atarget block to be decoded may need interpolation by a ½ pixel unit.Similarly, when a reference block is used to perform motion compensationby a ½ pixel unit, a target block may need interpolation by a ¼ pixelunit. Further, when a reference block is used to perform motioncompensation by a ¼ pixel unit, a target block may need interpolation bya ⅛ pixel (rational-pel) unit.

Motion compensation of the second transformation unit 260 may beperformed sequentially or at the same time along with resolutioncompensation. In order to perform motion compensation at the same timewith the resolution transformation operation, a combination filter isgenerated from a filter for motion compensation and a filter forresolution transformation and applied to a pixel value of a referenceblock. The following Table 2 illustrates an example of a combinationfilter to perform motion compensation and resolution transformation atthe same time in order to generate a ⅛ pixel when a magnitude of amotion vector decreases by half.

TABLE 2 MOTION KIND OF VECTOR FILTER RESOLUTION FILTER COEFFICIENTWIENER R¹ {1, −5, 116, 20, −5, 0} >> 7 INTERPOLATION Q¹ {2, −10, 104,40, −10, 2} >> 7 FILTER R² {3, −15, 92, 60, −15, 3} >> 7 (6-TAP) H {4,−20, 80, 80, −20, 4} >> 7 R³ {3, −15, 60, 92, −15, 3} >> 7 Q² {2, −10,40, 104, −10, 2} >> 7 R⁴ {1, −5, 20, 116, −5, 1} >> 7

In Table 2, filter coefficients of a combination filter to performmotion compensation and resolution transformation at the same time maybe determined according to a resolution of a motion vector needed formotion compensation.

The second transformation unit applies a plurality of adjacent integerpixels in the horizontal or vertical direction to a combination filterin Table 2 to generate a motion-compensated and resolution-transformedpixel value.

For example, with reference to FIG. 3, when generating aresolution-transformed pixel value corresponding to a ⅛ pixel positionedbetween the integer pixel C3 310 and the fractional pixel a 320 in thehorizontal direction, a combination filter may be applied to sixadjacent integer pixels by the following equationr=(C1−5C2+116C3+20C4−C5+0·C6)>>7.

Similarly, for motion interpolation and resolution transformation of a ½pixel and a ¼ pixel, the above combination filter is used to generate amotion-interpolated and resolution-transformed pixel value.

The synthesis unit 270 synthesizes a prediction block motion-compensatedand resolution-transformed in the second transformation unit 260 and adifferential block inverse-transformed and resolution-transformed in thefirst transformation unit 250, thereby generating aresolution-transformed target block. The resolution-transformed targetblock is synthesized with a plurality of other target blocks to generateone frame.

The inverse transformation 720 and the resolution transformation 710 ofthe first transformation unit 250 may be performed sequentially or atthe same time. When the inverse transformation 720 and the resolutiontransformation 710 are performed sequentially, an inverse quantizationof a differential image A_(4×4) in a 4×4 frequency received via theinverse quantization unit 230 is expressed by the following Equation 1.

$\begin{matrix}{{R_{4 \times 4} = {C_{i}^{T} \cdot A_{4 \times 4} \cdot C_{i}}},{C_{i} = {\begin{bmatrix}1 & 1 & 1 & 1 \\1 & {1/2} & {{- 1}/2} & {- 1} \\1 & {- 1} & {- 1} & 1 \\{1/2} & {- 1} & 1 & {{- 1}/2}\end{bmatrix}.}}} & \lbrack {{Equation}\mspace{14mu} 1} \rbrack\end{matrix}$

In Equation 1, R_(4×4) is an inverse-transformed 4×4 differential image,C, is an inverse transformation function used for various standards,e.g., H.264/AVC, and C_(i) ^(T) denotes a transposed matrix of C_(i).R_(4×4) generated by Equation 1 is resolution-transformed by thefollowing Equation 2.

$\begin{matrix}{{R_{2 \times 2} = {D_{2 \times 2}^{T} \cdot P_{2 \times 4} \cdot D_{4 \times 4} \cdot R_{4 \times 4} \cdot D_{4 \times 4}^{T} \cdot P_{4 \times 2} \cdot D_{2 \times 2}}},{P_{2 \times 4} = {\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0\end{bmatrix}.}}} & \lbrack {{Equation}\mspace{14mu} 2} \rbrack\end{matrix}$

In Equation 2, R_(2×2) is a resolution-transformed 2×2 differentialimage, D_(2×2) and D_(4×4) are a basis function to perform a discretecosine transform (DCT) operation on a 4×4 matrix and a 2×2 matrix,respectively, and P is a function for degree transformation. Equation 2illustrates a process of obtaining a differential block R_(2×2) byperforming resolution transformation and inverse transformation in thevertical and horizontal directions, respectively, based on the matrixR_(4×4).

The inverse transformation 720 and the resolution transformation 710 ofthe first transformation unit 250 may be performed at the same time.That is, the horizontal and vertical inverse transformation andresolution transformation in Equation 2 are combined to express a singleequation to obtain a differential block. The following Equation 3illustrates a process of transforming the received differential image inthe 4×4 frequency into a 2×2 spatial value using the single equation.

$\begin{matrix}{{R_{2 \times 2} = {K_{2 \times 4} \cdot A_{4 \times 4} \cdot G_{4 \times 2}}},{K_{2 \times 4} = \begin{bmatrix}1.4142 & 1.1152 & 0 & 0.0793 \\1.4142 & {- 1.1152} & 0 & {- 0.0793}\end{bmatrix}},{G_{4 \times 2} = {\begin{bmatrix}0.7071 & 0.5576 & 0 & 0.0396 \\0.7071 & {- 0.5576} & 0 & {- 0.0396}\end{bmatrix}^{T}.}}} & \lbrack {{Equation}\mspace{14mu} 3} \rbrack\end{matrix}$

In Equation 3, a matrix K_(2×4) is an equation to perform resolutiontransformation and inverse transformation in the vertical direction atthe same time in a DCT domain, and a matrix G_(4×2) is an equation toperform resolution transformation and inverse transformation in thehorizontal direction at the same time in the DCT domain.

In Equation 3, matrices K_(2×4) and G_(4×2) may use a coefficienttransformed into an integer to decrease a complexity of calculation. Forexample, when calculating using a coefficient 1.4142 in a 1×1 of amatrix K_(4×2), the coefficient is left-shifted by 8 bits before thecalculation to be transformed into an integer coefficient multiplied by2⁸. Then, after completing the calculation, the coefficient isright-shifted by 8 bits, thereby obtaining a final result divided by 2⁸.

The 2×2 differential image generated by the above process in the firsttransformation unit 250 is synthesized with a prediction block generatedby the second transformation unit 260 to generate aresolution-transformed target block.

The above image processing process may be used to predict a complexityof a bitstream and determine a decoding mode to reach a targetcomplexity.

FIG. 9 is a block diagram illustrating a decoding mode determinationdevice 900 according to an exemplary embodiment.

Referring to FIG. 9, the decoding mode determination device 900 of animage includes an extraction unit 910, a processing unit 920, and areconstruction unit 930.

The extraction unit 910 extracts information to predict a decodingcomplexity of an image from a bitstream of the image. The information topredict a decoding complexity of a received image may include at leastone of a frame height (FH) that is a resolution of an image, a framerate (FR) to indicate a number of received frames per second, aquantization parameter value, and the like.

The processing unit 920 predicts a complexity of an image usinginformation obtained by the extraction unit 910 and accommodates atarget complexity. Furthermore, the processing unit 920 calculates acomplexity ratio using the complexity of the image and the targetcomplexity to reach the target complexity. Moreover, the processing unit920 determines a decoding mode using the calculated complexity ratio.

Specifically, in order to predict a complexity of an image, theprocessing unit 920 uses an FH that is a resolution of the imageobtained by the extraction unit 910, an FR to indicate a number ofreceived frames per second, or a quantization parameter value. Here, theprocessing unit 920 calculates a value T_(e) to predict a complexity ofan image by performing a predetermined operation using a value obtainedby multiplying each predicated value by a predetermined magnitude value.

Furthermore, the processing unit 920 accommodates a target complexity.The target complexity is a value associated with time involved inreproducing a reconstructed image by an image processing apparatus andmay be determined based on a resource or a preset configuration of theimage processing apparatus. In further detail, the target complexity maybe calculated using a number of frames to reproduce a reconstructedimage per second. For example, when the number of frames reproduced persecond is 30 frames per second (FPS), a target complexity may becalculated by the following Equation 4.T _(c)=1000/FR _(i)(ms)  [Equation 4]

In Equation 4, T_(c) is a target complexity, and FR_(i) is a target FR.When 30 FPS is a target, a target complexity is a T_(c) of 1000/30(FPS)=33.333 ms. That is, a 30FPS bitstream is targeted to decode asingle frame in about 33.333 ms.

As described above, the processing unit 920 which calculates acomplexity of an image and a target complexity may obtain a complexityratio to determine a decoding mode using calculated complexities. Forexample, a complexity ratio to determine a decoding mode may be definedby the following Equation 5.

$\begin{matrix}{{{\underset{DMn}{\arg\;\min}\Delta\; T\mspace{14mu}{subject}\mspace{14mu}{to}\mspace{14mu}{\alpha \cdot \Delta}\; T} > \frac{T_{e} - T_{c}}{T_{e}}},} & \lbrack {{Equation}\mspace{14mu} 5} \rbrack\end{matrix}$where α is a constant determined experimentally. For example, when atarget complexity T_(c) is 33.333 ms, a predicted complexity of areceived image is 43.859 ms, and α is 1, a complexity ratio ΔT todetermine a decoding mode is 24% according to Equation 5.

The processing unit 920 may determine a decoding mode from modesprovided according to a complexity and image quality using a calculatedcomplexity ratio ΔT. A decoding mode is a mode to reconstruct an image,and may perform decoding or resolution transformation of a compressedimage according to a determined mode or perform decoding and resolutiontransformation of a compressed image at the same time.

A decoding mode may be determined by decoding and resolutiontransformation schemes. Table 3 below shows an example decoding modeavailable for decoding and resolution transformation of a compressedimage. Referring to Table 3, a decoding mode may be determined by acombination of a type of filter, a resolution of a motion vector, amasking value for inverse transformation, and a value to representwhether deblocking filtering is performed. In FIG. 3, a type of filteris a bilinear filter or a Wiener filter, a motion resolution unit uses ¼or ⅛ interpolation, a masking value for inverse transformation includesa 2×2 or 4×4 block. However, the above decoding mode is merely anexample to perform decoding or resolution transformation of an image,and a combination for a decoding mode according to another exemplaryembodiment may not be limited thereto and may vary.

TABLE 3 MOTION COMPENSATION RESOLUTION INVERSE TRANSFORMATION MOTIONVECTOR TRANSFORMATION DEBLOCKING FILTER RESOLUTION MASKING FILTERSELECTION BILINEAR (B) UP TO R-PEL/ 4 × 4/2 × 2 ON/OFF MODE 7-TAP FILTER(S) UP TO Q-PEL 13-TAP FILTER (W)

FIG. 10 illustrates examples of a decoding mode of a selectivecombination of the operations in Table 3 displayed according to acomplexity and image quality. In FIG. 10, an x coordinate denotes anormalized complexity ratio based on decoding and resolutiontransformation values which the functions associated with the abovedecoding mode are not applied to, and a y coordinate denotes a peaksignal-to-noise ratio (PSNR) to represent image quality.

In FIG. 10, a sign | denotes all results of combinations of theoperations in Table 3. Further, a sign ⊕ denotes a high efficiency modeamong the results of the combinations. In FIG. 10, the high efficiencymode includes a high quality mode 1010, a normal mode 1020, and a lowcomplexity and power save mode 1030.

The high quality mode 1010 is a decoding mode in which a filter is a13-tap Wiener filter, a resolution of a motion vector is a ⅛ pixel, amasking value for inverse transformation employs a 4×4 block, and adeblocking filter is not used. The normal mode 1020 is a decoding modein which a filter is a 7-tap Wiener filter, a resolution of a motionvector is a ⅛ pixel, a masking value for inverse transformation employsa 2×2 block, and a deblocking filter is not used. The low complexity andpower save mode 1030 is a decoding mode in which a filter is a bilinearfilter, a resolution of a motion vector is a ⅛ pixel, a masking valuefor inverse transformation employs a 4×4 block, and a deblocking filteris not used.

The processing unit 920 may select the high quality mode having a 24%complexity performance improvement in FIG. 10 when ΔT 1040 is calculatedinto 24% by the above Equation 5. Here, the image processing apparatusmay reconstruct a received image in the high quality mode.

The processing unit 920 may receive a mode used for decoding from a userin order to determine a decoding mode. For example, a plurality ofdecoding modes as described above are provided to a user, and the usermay select a desired decoding mode. Moreover, the processing unit 920may use a resource or a preset configuration of an image processingapparatus in order to determine a decoding mode.

The reconstruction unit 930 performs resolution transformation ordecoding of a received image, or performs resolution transformation ordecoding of a received image at the same time according to a decodingmode determined by the processing unit 920. Resolution transformationand decoding may be performed sequentially or at the same time. Anexample of the reconstruction unit 930 may be the image formingapparatus 200 in FIG. 2 or a part of the image forming apparatus 200 inFIG. 2 but is not limited thereto. The reconstruction unit 930 mayinclude other devices which are capable of performing resolutiontransformation or decoding.

FIG. 11 is a flowchart illustrating an image decoding process accordingto an exemplary embodiment.

Referring to FIG. 11, an image decoding apparatus according to anexemplary embodiment accommodates a reference block having a firstresolution and a differential block having the first resolution inoperation 1110. As described above, the reference block is a block towhich a target block having the first resolution in a frame to bedecoded refers in order to perform motion compensation. Further, thedifferential block having the first resolution is an image constitutedby a difference between the reference block and the target block havingthe first resolution.

Then, the image decoding apparatus according to the present exemplaryembodiment generates a prediction block having the first resolutionbased on a pixel value of the reference block having the firstresolution and transforms a resolution of the prediction block togenerate a prediction block having the second resolution in operation1120. In operation 1120, motion compensation may be performed by aninter pixel unit or a fractional pixel unit according to a resolution ofa motion vector. When performing motion compensation by a fractionalpixel unit, as described above, a pixel value of an adjacent area to thereference block having the first resolution may be used to generate apixel value of the prediction block having the first resolution.

Operation 1120 may include performing resolution transformation andperforming motion compensation, and the two operations may be performedat the same time.

When the two operations are performed at the same time, a combinationfilter may be used as described above. The combination filter isdetermined according to a resolution of a motion vector and a type offilter to transform the resolution. For example, different combinationfilters that may be used are an inter pixel, ½ pixel, ¼ pixel, or ⅛pixel, depending on a resolution of a motion vector. Moreover, thecombination filter may be determined based on a type of filter totransform the resolution.

The image decoding apparatus according to the present exemplaryembodiment transforms a resolution of the accommodated differentialblock having the first resolution to generate a differential blockhaving the second resolution in operation 1120. Operation 1120 may beperformed at the same time with an inverse transformation to generate aspatial value by inverse transformation of a differential image of thereference block having the first resolution and the target block havingthe first resolution.

In operation 1130, the differential block having the second resolutionand the prediction block having the second resolution generated inoperation 1120 are synthesized to generate a target block having thesecond resolution.

FIG. 12 is a flowchart illustrating an image decoding process accordingto another exemplary embodiment.

Referring to FIG. 12, in operation 1210, in order for a target blockhaving a first resolution in a frame being decoded to search for areference block for motion compensation, a magnitude of a motion vectorof the target block is transformed at a predetermined ratio. Here, thereference block is a block in a frame where a resolution has alreadybeen transformed and has a second resolution. The magnitude of thetransformed motion vector is proportionate to a ratio of a downsizing orupsizing resolution as described above.

In operation 1220, a prediction block having the second resolution isgenerated based on the reference block having the second resolutionindicated by the transformed motion vector. As described above, motioncompensation to generate the prediction block having the secondresolution may be performed by a more precise fractional pixel unit thanmotion compensation by the reference block having the second resolutionin order to increase accuracy of prediction.

In operation 1230, a differential block having the first resolution istransformed into a differential block having the second resolutionthrough a resolution transformation process. The differential blockhaving the first resolution is defined based on a difference between areference block having the first resolution and the target block havingthe first resolution. The operation of transforming a resolution of thedifferential block having the first resolution to generate thedifferential block having the second resolution may be performed at thesame time with a process of generating a spatial value by inversetransformation of a differential image of the first reference blockhaving the first resolution and the target block having the firstresolution.

In operation 1240, the prediction block having the second resolution andthe differential block having the second resolution generated inoperation 1220 and operation 1230, respectively, are synthesized togenerate a target block having the second resolution.

The above image processing process may be used for a decoding modedetermination process to perform reconstruction and resolutiontransformation of an image by predicting a complexity of a bitstream.

FIG. 13 is a flowchart illustrating a decoding mode determinationprocess according to an exemplary embodiment.

In operation 1310, a complexity of an image is predicted usinginformation extracted from a bitstream of the image. The information topredict a complexity of an image includes at least one of a measurementof time involved in decoding an image without resolution transformationin an offline state, an FH that is a resolution of an image, an FR toindicate a number of received frames per second, and a quantizationparameter value.

In operation 1320, a target complexity that is an aimed complexity isaccommodated. The target complexity is a value associated with timeinvolved in reproducing a reconstructed image by an image processingapparatus and may be determined based on a resource or a presetconfiguration of the image processing apparatus. In further detail, thetarget complexity may be calculated using a number of frames toreproduce one reconstructed image per second.

In operation 1330, a decoding mode used to perform resolutiontransformation and decoding of the image is determined based on thecomplexity of the image predicted in operation 1310 and the targetcomplexity of the image accommodated in operation 1320.

The decoding mode is a mode to reconstruct an image, and may performdecoding or resolution transformation of a compressed image based on adetermined mode, or may perform decoding and resolution transformationof a compressed image at the same time.

The decoding mode may be determined based on a resource or a presetconfiguration of the image forming apparatus. Alternatively, a pluralityof decoding modes are provided to a user, and the user may select adesired decoding mode.

The decoding mode may be determined using a difference in magnitudebetween the complexity of the image and the target complexity. Here, asdescribed above, a complexity ratio is calculated by Equation 5, and thecalculated complexity ratio is used to determine a decoding mode.

The decoding mode may be selected from among a plurality of modes, andeach of the modes may be a combination using at least one of a type offilter, a type of motion vector, a type of masking value for inversetransformation, and information about whether deblocking filtering isperformed, but is not limited thereto.

In operation 1340, resolution transformation and decoding of the imageare performed using the decoding mode determined in operation 1330.Resolution transformation and decoding may be performed sequentially orat the same time. An example to perform decoding may be the imageforming apparatus 200 in FIG. 2 or a part of the image forming apparatus200 in FIG. 2, but is not limited thereto. The example may include otherdevices which are capable of performing resolution transformation ordecoding.

The above-described exemplary embodiments may be recorded innon-transitory computer-readable media including program instructions toimplement various operations embodied by a computer. The media may alsoinclude, alone or in combination with the program instructions, datafiles, data structures, and the like. Moreover, one or more units of theimage processing apparatus 200 and the decoding mode determinationdevice 900 can include a processor or microprocessor executing acomputer program stored in a computer-readable medium.

Although a few exemplary embodiments have been shown and describedabove, exemplary embodiments are not limited thereto. Instead, it wouldbe appreciated by those skilled in the art that changes may be made tothese exemplary embodiments without departing from the principles andspirit of the disclosure, the scope of which is defined by the claimsand their equivalents.

What is claimed is:
 1. An image processing method of an image processingapparatus, the method comprising: obtaining a reference block a having afirst resolution and a differential block having the first resolution;transforming a resolution of a prediction block having the firstresolution and a resolution of the differential block having the firstresolution in order to generate a prediction block having a secondresolution and a differential block having the second resolution, theprediction block having the first resolution and the differential blockhaving the first resolution being generated based on a pixel value ofthe reference block having the first resolution; and synthesizing thegenerated prediction block having the second resolution and thegenerated differential block having the second resolution in order togenerate a target block having the second resolution.
 2. Anon-transitory computer readable recording medium having recordedthereon a program executable by a computer for performing the method ofclaim
 1. 3. An image processing method of an image processing apparatus,the method comprising: transforming a magnitude of a motion vector of atarget block having a first resolution at a predetermined ratio in orderto generate a transformed motion vector; generating a prediction blockhaving a second resolution based on a pixel value of a reference blockhaving the second resolution indicated by the transformed motion vector;transforming a resolution of a differential block having the firstresolution in order to generate a differential block having the secondresolution; and synthesizing the generated prediction block having thesecond resolution and the generated differential block having the secondresolution in order to generate a target block having the secondresolution.
 4. A non-transitory computer readable recording mediumhaving recorded thereon a program executable by a computer forperforming the method of claim
 3. 5. An image processing apparatuscomprising: a first transformation unit which transforms a differentialblock having a first resolution into a differential block having asecond resolution in order to generate the differential block having thesecond resolution; a second transformation unit which transforms aprediction block having the first resolution, generated based on a pixelvalue of a reference block having the first resolution, into aprediction block having the second resolution in order to generate theprediction block having the second resolution; and a synthesis unitwhich synthesizes the generated prediction block having the secondresolution and the generated differential block having the secondresolution in order to generate a target block having the secondresolution.
 6. An image processing apparatus comprising: a motion vectortransformation unit which transforms a magnitude of a motion vector of atarget block having a first resolution at a predetermined ratio in orderto generate a transformed motion vector; a second transformation unitwhich generates a prediction block having a second resolution based on apixel value of a reference block having the second resolution indicatedby the transformed motion vector; a first transformation unit whichtransforms a resolution of a differential block having the firstresolution in order to generate a differential block having the secondresolution; and a synthesis unit which synthesizes the generatedprediction block having the second resolution and the generateddifferential block having the second resolution in order to generate atarget block having the second resolution.
 7. An image processing methodof an image processing apparatus, the method comprising: predicting acomplexity of an image using information extracted from a bitstream ofthe image; obtaining a target complexity associated with reproducing theimage; determining a decoding mode used for resolution transformationand decoding of the image based on the complexity of the image and thetarget complexity; and performing the resolution transformation and thedecoding of the image according to the determined decoding mode, whereinthe information to predict the complexity of the image comprises atleast one of a resolution, a frame rate, and a quantization parameter ofthe image.
 8. The method of claim 7, wherein the target complexity is avalue determined based on at least one of a resource and a presetconfiguration of the image processing apparatus.
 9. The method of claim7, wherein the decoding mode is determined by selection of at least oneof a resource of the image processing apparatus, a preset configurationof the image processing apparatus, and an option of a user.
 10. Themethod of claim 7, wherein the determining the decoding mode comprisesselecting the decoding mode from among a plurality of decoding modes,and each of the plurality of decoding modes is associated with at leastone of a resolution transformation and a decoding scheme of the image.11. The method of claim 7, wherein the determining the decoding modecomprises determining the decoding mode based on a difference inmagnitude between the complexity of the image and the target complexity.12. The method of claim 7, wherein the decoding mode has a singlecombination among combinations including at least two of a type offilter, a resolution of a motion vector, a masking value for inversetransformation, and a value to represent whether deblocking filtering isperformed.
 13. A non-transitory computer readable recording mediumhaving recorded thereon a program executable by a computer forperforming the method of claim
 7. 14. An image processing apparatuscomprising: an extraction unit which extracts information from abitstream of an image to predict a complexity of the image; a processingunit which determines a decoding mode used for resolution transformationand decoding of the image according to the complexity of the imagepredicted based on the extracted information and a target complexityassociated with reproducing the image; and a reconstruction unit whichperforms the resolution transformation and the decoding of the imageaccording to the determined decoding mode, wherein the information topredict the complexity of the image comprises at least one of aresolution, a frame rate, and a quantization parameter of the image. 15.The image processing apparatus of claim 14, wherein the targetcomplexity is a value determined based on at least one of a resource anda preset configuration of the image processing apparatus.
 16. The imageprocessing apparatus of claim 14, wherein the decoding mode isdetermined by selection of at least one of a resource of the imageprocessing apparatus, a preset configuration of the image processingapparatus, and an option of a user.
 17. The image processing apparatusof claim 14, wherein the processing unit selects the decoding mode fromamong a plurality of decoding modes, and each of the plurality ofdecoding modes is associated with at least one of a resolutiontransformation and a decoding mode of the image.
 18. The imageprocessing apparatus of claim 14, wherein the processing unit determinesthe decoding mode based on a difference in magnitude between thecomplexity of the image and the target complexity.
 19. The imageprocessing apparatus of claim 14, wherein the decoding mode has a singlecombination among combinations including at least two of a type offilter, a resolution of a motion vector, a masking value for inversetransformation, and a value to represent whether deblocking filtering isperformed.
 20. The image processing apparatus of claim 14, wherein thereconstruction unit comprises: a first transformation unit which, whenthe image comprises a target block having a first resolution, obtains adifferential block having the first resolution and transforms thedifferential block into a differential block having a second resolutionin order to generate the differential block having the secondresolution; a second transformation unit which, when the image comprisesthe target block having the first resolution, obtains a prediction blockhaving the first resolution generated based on a pixel value of areference block having the first resolution and transforms theprediction block into a prediction block having the second resolution inorder to generate the prediction block having the second resolution; anda synthesis unit which, when the image comprises the target block havingthe first resolution, synthesizes the generated prediction block havingthe second resolution and the generated differential block having thesecond resolution in order to generate a target block having the secondresolution.
 21. The image processing apparatus of claim 14, wherein thereconstruction unit comprises: a motion vector transformation unitwhich, when the image comprises a target block having a firstresolution, transforms a magnitude of a motion vector of a target blockhaving the first resolution at a predetermined ratio in order togenerate a transformed motion vector; a second transformation unitwhich, when the image comprises the target block having the firstresolution, generates a prediction block having a second resolutionbased on a pixel value of a reference block having the second resolutionindicated by the transformed motion vector; a first transformation unitwhich, when the image comprises the target block having the firstresolution, transforms a resolution of a differential block having thefirst resolution in order to generate a differential block having thesecond resolution; and a synthesis unit which, when the image comprisesthe target block having the first resolution, synthesizes the generatedprediction block having the second resolution and the generateddifferential block having the second resolution in order to generate atarget block having the second resolution.